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Abstract 


This  grant  project  integrated  research  on  team  cognition,  temporal  dynamics,  and  storytelling 
towards  the  goal  of  improving  team  coordination  and  performance  in  distributed  decision  making 
teams.  Specifically,  the  three  overall  research  objectives  were:  1)  To  integrate  and  empirically 
investigate  multiple  types  of  team  cognition/macrocognition  (information  sharing,  team  mental 
models,  and  situational  awareness)  to  understand  their  differential  effects  on  team  outcomes  2)  to 
infuse  a  temporal  focus  into  the  study  of  team  cognition  and  team  outcomes  to  better  reflect  the 
context  of  organizational  and  military  teams  and  3)  to  investigate  how  storytelling  (complex 
form  of  metaphor)  can  be  used  as  a  team  training  intervention  to  improve  team  cognition  and 
collaboration.  Research  questions  were  tested  via  student  teams  performing  a  three-person  group 
task  called  NeoCITIES,  a  human-in-the-loop  scaled  world  simulation  which  is  set  within  the 
context  of  emergency  management  dispatch  center.  Results  provided  encouraging  evidence  that 
storytelling  and  reflexivity  interventions  may  help  overcome  the  collaborative  obstacles  faced  by 
team  members  in  distributed  environments,  particularly  when  administered  at  the  group  level. 
Infusing  a  temporal  focus  into  the  study  of  team  cognition  and  team  outcomes  proved  fruitful  in 
better  reflecting  the  time-based  context  of  many  organizational  and  military  teams. 
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GRANT  OVERVIEW 
Scientific  Objectives  and  Contributions 

This  grant  project  integrated  research  on  team  cognition,  temporal  dynamics,  and  storytelling 
towards  the  goal  of  improving  team  coordination  and  performance  in  distributed  decision  making 
teams.  Specifically,  the  three  overall  research  objectives  were:  1)  To  integrate  and  empirically 
investigate  multiple  types  of  team  cognition/macrocognition  (information  sharing,  team  mental 
models,  and  situational  awareness)  to  understand  their  differential  effects  on  team  outcomes  2)  to 
infuse  a  temporal  focus  into  the  study  of  team  cognition  and  team  outcomes  to  better  reflect  the 
context  of  organizational  and  military  teams  and  3)  to  investigate  how  storytelling  (complex 
form  of  metaphor)  can  be  used  as  a  team  training  intervention  to  improve  team  cognition  and 
collaboration.  This  program  of  research  expanded  team  cognition  research  into  previously 
unexplored  avenues  identified  as  potentially  fruitful  theoretically,  but  not  investigated 
empirically.  Because  failure  to  understand  “when”  can  seriously  jeopardize  final  team  outcomes, 
we  expand  the  definition  of  team  cognition  to  include  a  timework  dimension  in  addition  to  the 
traditional  teamwork  and  taskwork  content  domains.  In  addition,  as  storytelling  has  been  under¬ 
researched  in  the  team  literature,  we  investigate  how  the  powerful  benefits  of  ad  hoc  stories  can 
be  leveraged  as  a  team  training  tool  to  allow  team  members  to  get  on  the  same  page  more 
efficiently. 

Technical  Approach 

Research  questions  were  tested  via  student  teams  performing  a  three-person  group  task  called 
NeoCITIES,  a  human- in-the-loop  scaled  world  simulation  which  is  set  within  the  context  of 
emergency  management  dispatch  center,  and  designed  to  emulate  a  command  and  control 
computer  communication  (C4)  environment.  NeoCITIES  was  designed  to  allow  researchers  to 
closely  examine  team  behaviors  and  monitor  the  performance  outcomes  of  distributed  decision 
making  teams.  NeoCITIES  methodologically  offered  a  novel,  but  ideal,  interface  to  investigate 
multiple  aspects  of  team  cognition  due  to  its  adaptability,  flexibility,  and  controllability. 
NeoCITIES  allows  researchers  to  trim  away  the  complexities  and  confounds  of  a  more  complex 
task,  in  order  to  hone  in  on,  and  test  various  aspects  of  team  cognition  and  collaborative  decision 
making. 

Within  the  simulation,  teams  are  made  up  of  three  players,  who  are  each  assigned  to  the  role  of 
Fire/EMS,  Police,  or  Hazardous  Materials  (HazMat).  Each  role  is  assigned  3  unique  resources, 
each  with  varying  functions.  During  each  game,  players  were  tasked  with  solving  emergency 
management  events  by  allocating  appropriate  resources.  Scenarios  were  scripted  to  require  a 
dynamic  environment,  high  information  load,  the  need  to  distinguish  between  relevant  and 
irrelevant  information,  team  member  interdependence,  and  different  roles  with  unique 
knowledge  within  the  team. 
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Summary  of  Study  Findings  across  Four  Experiments: 

Across  studies,  results  revealed  that  various  types  of  team  cognition  impact  multiple  team 
performance  indices,  demonstrating  that  macrocognition  fosters  team  effectiveness.  Several 
types  of  team  cognition  were  positively  related  to  team  performance,  but  results  differed 
depending  on  the  type  of  team  cognition  (information  sharing,  shared  situation  awareness,  team 
mental  models)  and  the  team  outcome  assessed  (overall  team  performance,  timing). 

Infusing  a  temporal  focus  into  the  study  of  team  cognition  and  team  outcomes  proved  fruitful  in 
better  reflecting  the  time-based  context  of  many  organizational  and  military  teams.  Showing 
promise  for  future  research,  temporal  team  mental  models  were  distinct  from  taskwork  and 
teamwork  categories  and  added  unique  variance  beyond  these  traditional  measures. 

Results  provided  encouraging  evidence  that  storytelling  and  reflexivity  interventions  may  help 
overcome  the  collaborative  obstacles  faced  by  team  members  in  distributed  environments, 
particularly  when  administered  at  the  group  level.  Storytelling  as  an  intervention  did  in  fact 
increase  team  mental  model  similarity  over  a  non-story  format,  which  in  turn  increased  team 
performance.  However,  storytelling  is  most  useful  when  a  team  is  additionally  given  an 
opportunity  to  discuss  and  come  to  a  consensus  on  the  story’s  meaning  while  developing 
strategies  to  improve  future  performance.  In  addition,  allowing  team  members  to  communally 
reflect  upon  their  performance  and  strategies  is  more  effective  than  individual  reflection. 

Concise  Summary  of  Accomplishments 

To  our  knowledge,  we  were  the  first  to  empirically  examine  the  effect  of  planned  storytelling  as 
a  team  training  intervention  as  well  as  to  research  the  potential  additive  effects  of  group 
reflexivity  and  individual  reflexivity  on  team  mental  model  similarity.  In  addition,  we  pioneered 
the  operationalization  of  a  temporal  team  mental  model. 

We  conducted  four  programmatic  experimental  studies  (ranging  from  71  to  185  three-person 
teams  in  each  study)  examining  the  impact  of  multiple  interventions  (hidden  profile  task, 
storytelling,  group  reflexivity,  individual  reflexivity)  on  multiple  macro  cognition  constructs 
(information  sharing,  shared  situation  awareness,  team  mental  models)  and  team  performance. 

We  developed  NeoCITIES  3.0  and  3.1  simulation  test  beds,  which  allowed  for  a  higher  fidelity 
scoring  model  and  a  modem  technological  infrastructure  that  implemented  a  model  view ' 
controller  framework  for  modular  interface.  NeoCITIES  3.0  and  3.1  enabled  the  empirical 
investigation  of  under-researched  constructs  such  as  storytelling  and  temporal  mental  models  as 
well  as  integration  across  macrocognitive  concepts  such  as  situational  awareness  and  information 
sharing. 

In  terms  of  output,  we  produced  six  publications,  fourteen  refereed  conference  presentations,  two 
Master’s  theses,  and  one  dissertation.  One  manuscript  is  currently  under  review  at  a  peer 
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reviewed  journal  with  several  others  in  preparation  for  journal  submission.  These  manuscripts 
are  based  on  the  collective  sum  total  of  research  conducted  under  this  grant.  One  of  the 
presentations  was  featured  as  a  top-rated  poster  at  the  201 1  Society  for  Industrial/Organizational 
Psychology  annual  conference  and  a  finalist  for  the  best  student  submission  award.  The 
Appendix  contains  a  full  listing  of  grant  output. 


THEORETICAL  BACKGROUND 
Research  Objectives  and  Contributions 

Increasingly,  work  is  carried  out  by  teams  consisting  of  diverse  members  who  have  never 
interacted  previously,  but  are  required  to  perform  complex,  dynamic  tasks  in  novel  and  time- 
pressurized  environments  (e.g.,  Tannenbaum,  Mathieu,  Salas,  &  Cohen,  2012).  Given  the 
difficulty  of  this  context,  the  issue  of  how  to  enhance  team  collaboration  and  performance  is 
increasingly  salient  in  organizational  and  military  settings.  Although  already  a  considerable 
challenge  in  co-located  teams,  getting  members  “on  the  same  cognitive  page  is  even  more 
difficult  when  team  members  are  geographically  separated  (e.g.,  Martins,  Gilson,  &  Maynard, 
2004).  This  grant  project  integrated  research  on  team  cognition ,  temporal  dynamics,  and 
storytelling  towards  the  goal  of  improving  team  coordination  and  performance  in  distributed 
decision  making  teams.  Specifically,  the  three  overall  research  objectives  were: 

•  To  integrate  and  empirically  investigate  multiple  types  of  team  cognition/macrocognition 
(information  sharing,  team  mental  models,  and  situational  awareness)  to  understand 
their  differential  effects  on  team  outcomes 

•  To  infuse  a  temporal  focus  into  the  study  of  team  cognition  and  team  outcomes  to  better 
reflect  the  context  of  organizational  and  military  teams 

•  To  investigate  how  storytelling  (complex  form  of  metaphor)  can  be  used  as  a  team 
training  intervention  to  improve  team  cognition  and  collaboration 

These  research  objectives  and  study  results  contribute  to  the  team  literature  in  multiple  ways. 
First,  learning  how  various  forms  of  team  cognition  integratively  influence  team  outcomes  has 
been  identified  as  a  critical  research  need  (e.g.,  Salas  &  Wildman,  2009;  Mohammed,  Ferzandi, 
&  Hamilton,  2010).  Moreover,  empirical  work  is  clearly  needed  to  achieve  better  understanding 
of  the  processes  underlying  macrocognition  and  to  establish  measurement  tools  (e.g.,  Fiore, 
Smith-Jentsch,  Salas,  Letsky,  &  Warner,  2010).  As  such,  we  empirically  investigate  information 
sharing,  situation  awareness,  and  team  mental  models  in  the  same  set  of  studies,  allowing  for  a 
deeper  understanding  of  their  differential  effects.  Second,  as  time  has  been  identified  as  “perhaps 
the  most  neglected  critical  issue”  in  team  research  (Kozlowski  &  Bell,  2003,  p.  364)  and  team 
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cognition  measures  have  been  deemed  temporally  deficient  (Mohammed,  Tesler,  &  Mohammed, 
2012),  we  infused  time  into  assessments  of  information  sharing,  situational  awareness,  and  team 
mental  models.  As  such,  answer  numerous  calls  to  sharpen  the  temporal  lens  used  in  conducting 
team  studies  (e.g.,  Ancona,  Goodman,  Lawrence,  &  Tushman,  2001;  Mohammed,  Hamilton,  & 
Lim,  2009). 

Third,  because  much  of  the  work  on  storytelling  is  conceptual  (e.g.,  Denning,  2001),  and 
empirical  work  has  been  largely  focused  at  the  individual  level  of  analysis  (e.g.,  Ang  &  Rao, 
2008),  its  applicability  in  team  settings  is  under-researched.  To  our  knowledge,  we  are  the  first  to 
investigate  how  the  powerful  benefits  of  ad  hoc  stories  can  be  leveraged  as  a  team  training  tool 
to  allow  team  members  to  achieve  higher  levels  of  shared  understanding  and  thereby  perform 
more  effectively.  Fourth,  through  the  integration  of  team  cognition,  temporality,  and  storytelling, 
we  expand  these  literatures  into  previously  unexplored  avenues  identified  as  potentially  fruitful 
theoretically,  but  not  investigated  empirically.  For  example,  answer  previous  research  calls  to 
broaden  the  list  of  empirically  verified  antecedents  of  team  mental  models  (e.g.,  Mohammed  et 
al.,  2010)  by  examining  the  effects  of  storytelling  on  team  cognition.  Fifth,  beyond  these 
theoretical  contributions,  NeoCITlES  methodologically  offers  a  novel,  but  ideal,  interface  to 
investigate  multiple  aspects  of  team  cognition  because  it  is  easily  adaptable  and  designed  to  test 
team  cognition  and  collaborative  decision  making  processes. 

Team  Cognition  &  Macrocognition 

Team  cognition  is  a  general  term  referencing  the  collective  cognitions  of  a  group  (Salas  &  Fiore, 
2004).  Over  the  past  decade,  the  amount  of  team  cognition  research  has  increased  substantially 
across  disciplinary  boundaries  (e.g.,  Badke-Schuab,  Neumann,  Lauche,  &  Mohammed,  2007; 
Fiore  &  Salas,  2006;  Undre,  Sevdalis,  Healey,  Darzi,  &  Vincent,  2006).  In  addition  to  being 
identified  as  one  of  the  hallmarks  of  expert  teams  (Salas,  Rosen,  Burke,  Goodwin,  &  Fiore, 
2006),  and  a  recent  meta-analysis  found  that  team  cognition  positively  predicted  team 
motivation,  processes,  and  performance  (DeChurch  &  Mesmer-Magnus,  2010). 

More  recently,  the  notion  of  macrocognition  has  been  advocated  as  a  promising  mechanism  to 
enable  teams  to  achieve  more  efficient  and  effective  collaboration.  Macrocognition  is  defined  as 
the  “internalized  and  externalized  high-level  mental  processes  employed  by  teams  to  create  new 
knowledge  during  complex,  one-of-a-kind,  collaborative  problem  solving”  (Letsky  &  Warner, 
2008,  p.  7).  Macrocognition  is  viewed  as  a  specific  exemplar  of  the  more  general  area  of  team 
cognition  research  (Fibre,  Rosen,  Smith-Jentsch,  Salas,  Letsky,  &  Warner,  2010).  Whereas  team 
cognition  has  focused  on  rule-based  performance  in  familiar  contexts,  macrocognition 
emphasizes  knowledge-based  performance  in  novel  situations  (Fiore,  Rosen  et  al.,  2010).  As 
such,  macrocognition  was  an  important  theoretical  focus  of  the  grant  research,  which  sought  to 
understand  and  improve  team  performance  in  unique,  complex,  dynamic,  and  time-sensitive 
environments. 
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According  to  Fiore,  Rosen,  and  colleagues  (2010),  the  team  collaborative  stages  of  the 
macrocognition  model  include  knowledge  construction,  problem  model  development,  team 
consensus,  and  outcome  evaluation  and  revision.  Our  research  especially  focuses  on  the  problem 
model  development  stage,  reflecting  how  a  team  arrives  at  a  shared  understanding  of  the  problem 
and  viable  solutions.  In  addition,  we  measure  multiple  macrocognitive  processes,  including 
individual  knowledge  building,  team  knowledge  building,  internalized  team  knowledge, 
externalized  team  knowledge,  and  team  problem  solving  outcomes.  However,  our  research 
particularly  features  team  knowledge  building  and  internalized  team  knowledge  in  that  we 
measure  team  information  exchange,  team  knowledge  exchange,  team  mental  model  similarity 
and  accuracy,  as  well  as  shared  situation  awareness.  Team  problem  solving  outcomes  are  also 
well  represented  in  our  research,  as  the  NeoCITIES  simulation  has  the  capability  to  capture 
multiple  performance  metrics  that  assess  the  effectiveness  as  well  as  the  efficiency  of  chosen 
solutions. 

As  both  team  cognition  and  macrocognition  are  broad  terms  that  include  specific  exemplars,  our 
decision  to  focus  on  team  situational  awareness,  temporal  mental  models,  and  information  shared 
was  based  on  the  current  models  of  macrocognition.  For  example,  components  of  team 
knowledge  building  include  information  sharing,  team  situation  awareness,  and  team  mental 
models  (Letsky  &  Warner,  2008;  Warner  &  Letsky,  2008).  In  addition,  the  sharing  of  unique 
information  has  been  identified  as  a  critical  macrocognitive  process  (McComb,  2008).  As  such, 
this  project  furthers  understanding  of  the  processes  underlying  macrocognition  in  teams  as  well 
as  their  relationship  to  other  constructs.  In  addition,  these  constructs  are  relevant  within  the 
NeoCITIES  platform. 

Temporal  Dynamics  &  Temporal  Team  Mental  Models 

Because  increased  attention  to  temporal  matters  has  been  identified  as  a  key  agenda  for  future 
group  research  (e.g.,  Eisenhardt,  2004;  McGrath  &  Tschan,  2004),  we  emphasized  multiple 
dimensions  of  temporality  in  the  current  research.  For  example,  the  scenarios  within  the 
NeoCITIES  simulation  were  designed  to  emphasize  temporal  sequencing  and  pacing.  In 
addition,  as  the  role  of  time  has  been  largely  downplayed  in  team  cognition  research 
(Mohammed  et  al.,  2012),  we  add  a  temporal  referent  to  many  of  our  measures.  As  we  discuss 
below,  special  attention  was  given  to  expanding  the  conceptualization  of  team  mental  models  to 
include  time. 

Team  mental  models  are  defined  as  mental  representations  of  the  key  elements  within  a  team’s 
relevant  environment  that  are  both  understood  and  shared  across  team  members  (Mohammed  & 
Dumville,  2001).  Teams  whose  members  share  models  of  both  taskwork  (e.g.,  equipment, 
performance  requirements)  and  teamwork  (interpersonal  interaction  requirements)  domains  are 
better  positioned  to  anticipate  the  needs  and  actions  of  other  members,  thereby  increasing  team 
performance  (e.g.,  Marks,  Zaccaro,  &  Mathieu,  2000;  Mathieu,  Heffner,  Goodwin,  Salas,  & 


8 


Cannon-Bowers,  2000).  Researchers  from  various  areas  have  become  increasingly  confident  that 
one  of  the  keys  to  team  effectiveness  lies  within  the  team  mental  model  (e.g.,  DeChurch  & 
Mesmer-Magnus,  2010;  Mohammed  et  al.,  2010). 

Although  temporal  dynamics  clearly  play  a  key  role  in  team  mental  models,  the  role  of  time  in 
both  conceptualization  and  measurement  has  been  largely  ignored  in  existing  research.  Rather, 
studies  have  emphasized  task  procedures  (“what”)  and  team  interaction  patterns  (“how”  and 
“who”)  (e.g.,  Mathieu  et  al.,  2000)  as  two  broad  categories  of  mental  model  content.  However, 
.when  team  members  disagree  about  deadlines  and  the  pace  by  which  tasks  should  be  completed, 
coordination  and  performance  problems  can  result.  Therefore,  establishing  and  maintaining 
congruence  in  team  members’  temporal  perceptions  is  a  non-trivial  task.  Since  failure  to 
understand  “when”  can  seriously  jeopardize  final  team  outcomes,  we  add  temporality  to  the 
construct  of  team  mental  models.  Specifically,  temporal  mental  models  are  defined  as  agreement 
among  group  members  concerning  deadlines  for  task  completion,  the  pacing  or  speed  at  which 
activities  take  place,  and  the  sequencing  of  tasks  (Mohammed  et  al.,  2012).  For  more  information 
about  the  conceptual  foundation  for  integrating  time  and  team  cognition,  see  Mohammed  et  al. 
(2012). 

Metaphors  and  Storytelling 

Given  that  team  cognition  and  macrocognitive  processes  facilitate  team  effectiveness  (e.g., 
DeChurch  &  Mesmer-Magnus,  2010;  Fiore,  Rosen  et  al.,  2010),  interventions  should  be 
developed  to  enhance  the  development  of  shared  understanding  in  teams.  As  such,  we  proposed 
that  storytelling  (complex  form  of  metaphor),  as  a  powerful  means  of  understanding  and 
structuring  reality,  will  facilitate  team  members  achieving  higher  levels  of  shared  consensus. 
Below,  we  briefly. review  the  literature  on  metaphors  and  storytelling. 

Metaphors  are  powerful  tools  of  social  constructions  that  imply  “a  way  of  thinking  and  a  way  of 
seeing”  (Morgan,  1986,  p.  12).  A  primary  purpose  of  metaphors  is  to  enable  us  to  understand 
aspects  of  reality  and  structure  behavior  (Stutman  &  Putnam,  1994).  By  analogical  extension, 
metaphors  carry  ideas  from  one  domain  to  another  in  which  the  ideas  are  not  immediately 
applicable  (Black,  1962).  Sensemaking  is  enhanced  by  comparing  new  concepts  that  have  not 
been  directly  or  indirectly  experienced  to  what  is  already  known.  Metaphors  play  an  important 
role  in  surfacing  tensions  and  ambiguities  inherent  in  constructs.  In  addition  to  altering 
perceptions,  metaphors  also  structure  behavior.  By  improving  understanding  regarding  an 
unknown  and  unfamiliar  phenomenon,  metaphors  shape  future  actions  (Schultz  &  Orlikowski, 
2001). 

Categories  of  metaphors  include  spatial  (capture  a  direction  that  is  typically  associated  with 
positive  and  negative  attributions),  ontological  (utilize  concrete  substances  or  objects  to  describe 
abstract  experiences  like  states  and  emotions),  metonymy  (capture  the  whole  in  terms  of  a 
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constitutive  part),  personification  (apply  human  traits  to  inanimate  objects),  and  structural  (match 
structural  requirements  of  the  phenomenon  with  those  of  the  metaphor)  (Lakoff  &  Johnson, 

1980).  Numerous  examples  of  the  use  of  metaphor  can  be  found  in  psychological  and 
organizational  literatures.  For  example,  organizations  have  been  described  in  terms  of  machines, 
organisms,  and  brains  (Morgan,  1986).  In  identifying  several  overarching  metaphors  of 
virtuality,  Schultz  &  Orlikowski  (2001)  described  virtual  organization  as  platform  (architecture), 
space  (location),  bits  (resource),  community  (governance),  and  a  network  of  relationships 
(identity).  In  addition,  metaphor  has  been  used  extensively  in  group  therapy  (e.g.,  Sunderland, 
1997)  and  to  describe  group  process  (e.g.,  Cowell,  1972). 

Because  they  transfer  meaning  across  conceptual  domains,  metaphors  are  especially  valuable 
when  exploring  complex,  novel,  and  poorly  understood  phenomena  (Klagge,  1997).  According 
to  Lakoff  and  Johnson  (1980),  metaphors  are  “one  of  the  most  important  tools  for  trying  to 
comprehend  partially  what  cannot  be  comprehended  totally”  (p.  193).  In  addition,  metaphors 
play  a  critical  role  in  early  stages  of  theory  development,  before  specific  hypotheses  are 
articulated  (Conmelissen,  2005).  Given  the  complexity  of  macro  cognitive  processes  as  well  as  its 
formative  stage  of  research  development,  examining  the  role  of  metaphors  in  enabling  teams  to 
get  on  the  same  cognitive  page  more  efficiently  and  effectively  is  particularly  timely. 

As  a  more  complex  version  of  metaphors,  storytelling  is  one  of  the  oldest  methods  for 
communicating  knowledge  across  people  (e.g.,  Bal,  1997;  Denning,  2001).  The  purpose  of  a 
story  or  narrative  is  to  convey  complicated  ideas  in  an  intriguing  and  clearly  understood  manner 
(e.g.,  Klein,  1998).  Several  authors  have  proposed  that  storytelling  can  improve  team  learning 
and  performance  (e.g.,  Bartel  &  Garud,  2009;  Denning,  2001;  Fiore  et  al.,  2009).  For  example, 
Fiore,  McDaniel,  and  Jentsch  (2009)  state  that  “stories  can  be  a  form  of  contextual  glue  that 
becomes  the  basis  for  an  ensuing  dialogue  as  teams  evolve  over  space  and  time  in  information 
spaces  mediated  by  both  virtual  and  physical  cues.”  We  explored  the  use  of  storytelling  as  a 
planned  team  training  tool  because  storytelling  has  been  proposed  to  be  effective  in  distributed 
teams  (Fiore  et  al.,  2009),  but  not  empirically  verified.  Given  that  storytelling  may  be  especially 
valuable  when  exploring  complex,  novel,  and  poorly  understood  phenomena,  it  may  prove 
especially  useful  in  aiding  geographically  distributed  members  to  achieve  shared  understanding. 
For  more  information  about  storytelling  as  a  training  intervention,  see  Mancuso,  Parr,  McMillan, 
Tesler,  McNeese,  Hamilton,  &  Mohammed  (2011),  Tesler,  Mohammed,  Hamilton,  Mancuso, 
Parr,  MacMillan,  &  McNeese  (201 1),  and  Tesler,  Mohammed,  Mancuso,  Hamilton,  &  McNeese 
(2012). 


THE  NeoCITIES  TEAM  SIMULATION 

NeoCITIES  is  a  human-in-the-loop  scaled  world  simulation  designed  to  allow  researchers  to 
closely  examine  team  behaviors  and  monitor  the  performance  outcomes  of  spatially  distributed 
decision-making  teams  (McNeese,  Bains,  Brewer,  Brown,  Connors,  Jefferson,  Jones  &  Terrell, 
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2005).  Abstracted  from  the  original  CITIES  task  (Wellens  &  Ergener,  1988),  NeoCITIES  uses 
information  processing  theories  and  knowledge  elicitation  research  (Wellens  &  Ergener,  1 988)  to 
examine  the  role  of  cognitive  processes  in  facilitating  knowledge  acquisition  and  transfer 
(McNeese,  2000).  NeoCITIES  features  team  resource  allocation  problems  designed  to  emulate 
crisis  management  of  a  city’s  emergency  services  in  a  virtual  environment.  In  the  early  stages  of 
its  creation,  realistic  scenarios  were  developed  from  interviews  with  intelligence  analysts  and 
fieldwork  in  emergency  crisis  centers  (McNeese  et  al.,  2005). 

The  newest  versions  of  NeoCITIES  (3.0  and  3.1)  are  web-based  applications  based  on  instant 
chat  technology  and  an  interactive  interface.  Three  person  teams  play  the  roles  of  police, 
fire/EMS,  and  hazardous  materials  (hazmat).  Each  unit  is  responsible  for  different  resources.  For 
example,  the  fire  unit  has  ambulances,  fire  investigators,  and  trucks  at  their  disposal.  Team 
members  are  tasked  with  dispatching  the  relevant  type  and  number  of  resources  to  emergency 
management  events  that  arise.  When  faced  with  a  new  scenario,  participants  must  decide  the 
severity  of  incoming  events,  what  resources  are  needed,  and  how  many  resources  are  needed, 
and  whether  assistance  is  required  of  other  team  members.  NeoCITIES  allows  for  both 
qualitative  (chat  logs)  and  quantitative  (performance  scores  and  other  behavioral  metrics)  data 
collection.  For  more  information  about  the  NeoCITIES  simulation,  refer  to  Hellar  &  McNeese 
(2010)  and  Hamilton,  Mancuso,  Minotra,  Hoult,  Mohammed,  Parr,  Dubey,  MacMillan  & 
McNeese  (2010). 

Rationale  for  Selecting  NeoCITIES 

There  are  several  reasons  why  NeoCITIES  is  well  suited  to  test  the  research  objectives 
articulated  above.  First,  it  is  designed  to  test  team  collaborative  decision  making  process, 
knowledge  acquisition  and  knowledge  management  within  a  command  and  control  computer- 
mediated  communication  environment.  As  such,  it  is  the  ideal  interface  to  investigate 
macrocognitive  processes  such  as  information  sharing,  shared  situation  awareness,  and  team 
mental  models.  Second,  emergency  crisis  management  offers  an  excellent  real  world  domain  in 
which  meets  the  requirements  of  a  complex  system  articulated  by  Rouse,  Cannon-Bowers,  & 
Salas  (1992),  including  a  highly  emergent,  dynamic,  and  complex  environment,  ambiguous  and 
uncertain  information,  as  well  as  individuals  with  differing  roles  needing  to  interact  and  share 
information  as  a  team.  Third,  as  an  adaptable  interface,  NeoCITIES  can  be  scripted  to  represent 
routine  (e.g.,  law  enforcement)  and  non-routine  (e.g.,  terrorism)  scenarios  ranging  from 
individual  events  to  highly  interdependent  occurrences  that  escalate  when  resources  are  not 
properly  allocated.  Therefore,  the  simulation  is  scalable  and  flexible  enough  to  accommodate  the 
new  scenario  development  that  was  needed  to  test  research  questions  involving  macrocognition, 
temporality,  and  storytelling.  Fourth,  objective  dependent  variables  are  easily  collected, 
including  team  performance  (quantitative  score  indicating  overall  progress),  time  to  decision, 
decision  errors,  and  number  of  failed  events.  Fifth,  participants  find  NeoCITIES  to  be  an 
engaging,  realistic  exercise  which  maintains  interest  and  high  levels  of  student  participation. 
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The  NeoCITIES  3.1  User  Interface 

The  NeoCITIES  3.1  interface  is  made  up  of  five  primary  components:  the  chat  panel,  the 
dispatch  panel,  the  team  monitor,  the  unit  monitor,  and  the  event  tracker  (see  Figure  1).  The  chat 
panel  (A)  allows  users  to  communicate  and  coordinate  with  each  other.  The  chat  panel  is  a 
necessary  feature  in  the  interface  because  several  of  the  events  in  NeoCITIES  3.1  require 
coordination  among  the  various  roles  in  the  simulation.  The  dispatch  panel  (B)  indicates  which 
resources  are  currently  available.  To  dispatch  resources,  the  user  drags  the  desired  resources  into 
a  box  and  presses  a  dispatch  button.  The  resources  are  then  dispatched  to  the  selected  event  and 
are  no  longer  available.  Resources  are  returned  to  the  dispatch  panel  when  an  event  completes  or 
a  resource  is  recalled.  The  team  monitor  (C)  provides  information  on  the  event  the  participant’s 
teammates  are  currently  working,  along  with  the  type  and  number  of  resources  available  to  them. 
This  information  is  useful  for  helping  team  members  collaborate  and  coordinate  with  each  other. 
The  unit  monitor  (D)  provides  participants  with  information  on  their  units’  status  and  location. 
The  unit  monitor  also  provides  feedback  reports  on  the  events  to  which  the  participant  has 
responded.  The  reports  indicate  whether  or  not  the  unit  was  able  to  help  with  a  specific  event  and 
whether  more  resources  are  needed.  The  participant  may  recall  a  resource  if  it  is  unable  to  help 
resolve  an  event.  The  event  tracker  (E)  is  divided  into  two  parts:  a  list  of  current  events  and  a  list 
of  closed  events  on  which  no  more  action  may  be  taken.  In  this  section,  participants  can  see  the 
name,  description,  status,  and  severity  of  current  events.  The  tracker  is  also  used  to  select  the 
events  to  which  resources  are  dispatched. 
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Figure  1:  TheNeoCITIES  3.1  User  Interface 

A=  Chat  Panel;  B=Dispatch  Panel;  C=Team  Monitor;  D=Unit  Monitor;  E=Event  Tracker 
Improvements  in  NeoCITIES  over  Years  1-3  of  the  Grant 

The  methodological  focus  of  the  first  year  of  the  grant  was  the  development  of  NeoCITIES  3.0 
and  the  set  up  of  the  client  server  architecture,  which  was  under  construction  for  more  than  nine 
months,  requiring  over  10,000  lines  of  code  and  over  a  month  of  pilot  testing  the  new  interface. 
As  this  simulation  was  the  platform  upon  which  all  of  our  experiments  were  conducted,  the  time 
needed  to  complete  the  development  and  piloting  of  NeoCITIES  was  a  worthy  investment  to 
ensure  ecological  and  internal  validity. 

NeoCITIES  3.0  was  built  using  Web  2.0  technologies,  and  was  designed  as  a  more  flexible 
research  tool  and  to  have  a  more  powerful  graphical  human-computer  interface.  Specifically, 
NeoCITIES  3.0  allowed  for  a  higher  fidelity  scoring  model,  data  logging,  and  a  modem 
technological  infrastructure  that  implemented  a  model  view  controller  framework  for  modular 
interface.  In  addition,  it  allowed  for  an  adaptive  interface  that  was  scalable  and  flexible  enough 
to  accommodate  new  scenario  development.  Figure  2  shows  a  screenshot  of  the  NeoCITIES  3.0 
interface. 
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Figure  2:  The  NeoCITIES  3.0  User  Interface 

In  the  second  year  of  the  grant  (in  preparation  for  Studies  2  and  3),  further  refinements  to 
NeoCITIES  were  made  to  upgrade  the  interface  and  add  new  data  collection  mechanisms, 
resulting  in  the  most  recent  3.1  version.  For  example,  we  created  a  video-based  training  (auditory 
as  well  as  visual)  to  introduce  more  experimental  control  over  the  self-paced  PowerPoint  slides 
(text-based  only)  utilized  in  the  first  study.  On  the  server,  we  improved  score  reporting  by 
building  the  capability  to  log  more  rich  behavior  metrics,  including  reaction  times, 
communication  patterns,  and  order,  to  name  a  few.  On  the  client  computers,  time  was  changed  to 
a  12-hour  format  rather  than  the  timer  format  used  previously.  Given  the  importance  of 
temporality  in  our  studies,  this  was  an  important  and  needed  change.  In  addition,  time  stamps 
were  added  on  the  chat  messages,  which  helped  with  coding  chat  logs.  Furthermore,  if  the 
experimenter. had  already  “initialized”  the  game  and  the  player  loaded  up  their  screen  late,  it 
would  direct  them  to  the  login  screen. 
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Based  on  the  analysis  of  data  in  the  first  three  studies,  several  modifications  again  were  made  to 
the  NeoCITlES  simulation  in  the  third  year  of  the  grant  (in  preparation  for  Study  4).  For 
example,  the  interface  was  reorganized  to  allow  for  easier  participant  navigation  and  clarity,  as 
shown  in  Figure  3.  Measures  that  were  previously  in  paper  and  pencil  form  were  transferred  to 
an  online  format  within  NeoCITlES,  including  SAGAT  (freeze-probe  measure  of  situational 
awareness)  and  information  briefings.  Improvements  were  also  made  to  the  team  mental  model 
measures.  Significant  time  was  also  invested  in  altering  the  simulation  scenarios  to  enhance  the 
temporal  performance  measures  in  the  game. 
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Figure  3:  Modifications  to  the  NeoCITlES  User  Interface  Used  in  Study  4 


Table  1  summarizes  the  history  of  the  NeoCITlES  simulation.  For  additional  information  about 
the  development  of  NeoCITlES  in  comparison  to  other  simulations,  see  McNeese  and  Pfaff 
(2012). 
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Table  1 :  Summary  of  the  History  of  the  NeoCITIES  simulation 
Scenario  Development  in  NeoCITIES 

In  addition  to  the  effort  required  to  build  the  NeoCITIES  simulation,  creating  the  scenarios 
within  the  simulation  required  a  large  investment  of  time,  thought,  and  creativity.  Emergency 
events  were  scripted  around  the  university  campus  to  increase  realism  and  relevance  to  student 
participants  (e.g.,  student  rioting,  bomb  threats  during  exams,  intoxication,  apartment  fires,  and 
fumes  from  the  chemistry  building).  For  studies  1-3,  the  first  scenario  involved  a  football  theme, 
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and  the  second  scenario  involved  a  finals  exam  week  theme.  For  study  4,  the  second  scenario 
was  altered  to  remove  some  complexity  involving  hidden  profile  information,  based  on  piloting 
feedback  and  little  performance  variability  on  certain  events.  Events  varied  in  their  severity  (e.g., 
trash  can  fire  versus  a  chemical  tanker  truck  collision). 

Scenarios  were  written  to  require  a  dynamic  environment,  high  information  load,  the  need  to 
distinguish  between  relevant  and  irrelevant  information,  and  different  roles  with  unique 
knowledge  within  the  team.  Some  events  were  independent  in  that  they  only  required  resources 
from  one  unit  (e.g.,  only  EMS  was  required  to  revive  a  collapsed  student  suffering  from 
dehydration).  However,  other  events  were  interdependent  in  that  they  required  resources  from 
multiple  roles  (e.g.,  police,  fire,  and  hazmat  units  were  needed  to  respond  to  a  large  fire  that  may 
have  been  chemically  induced  and  involved  the  possibility  of  arson).  Some  interdependent 
events  required  a  response  from  two  units  and  others  required  all  three.  Additionally,  temporal 
requirements  were  infused  in  scenarios  by  prescribing  that  teammates  respond  on  the  scene  of 
events  within  a  particular  time  frame  or  in  a  certain  sequence  (e.g.,  Police  first,  fire/EMS  second, 
and  hazmat  third).  In  the  third  year  of  the  grant  (in  preparation  for  Study  4),  simulation  scenarios 
were  rewritten  and  piloted  to  improve  temporal  measurement  in  the  game. 

DESCRIPTION  OF  PROCEDURES  AND  MEASURES  COMMON  ACROSS  ALL  FOUR 

EXPERIMENTS 


Laboratory  Set-Up 

Data  collection  necessitated  the  set-up  of  two  parallel  laboratory  infrastructures  in  psychology  as 
well  as  information  sciences  and  technology.  In  each  lab,  six  separate  computers  were  isolated 
from  each  other  via  dividers  to  mimic  distributed  teams.  Three  computer  stations  were  in  a  row 
on  one  side,  with  another  three  computer  stations  across  from  the  first  row  on  the  other  side. 
Therefore,  each  lab  could  run  two  3-person  teams  through  the  NeoCITIES  simulation  at  one 
time.  There  were  also  two  server  computers  used  by  experimenters  at  the  front  of  the  first  two 
rows. 
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Figure  4:  Pictures  of  the  Psychology  and  Information  Science  and  Technology  Data  Collection 
Labs 

Participants 

Student  participants  were  obtained  from  undergraduate  information  sciences,  psychology, 
business,  and  human  development  classes  as  well  as  the  psychology  subject  pool.  Students 
received  course  or  extra  credit  for  their  participation. 

General  Experimental  Procedure 

After  random  assignment  to  a  three-person  team  and  a  member  role  (Police,  Hazmat,  or 
Fire/EMS),  students  were  first  trained  on  the  mechanics  of  playing  the  NeoCITIES  simulation. 
This  training  covered  how  to  review  event  descriptions  and  allocate  resources  to  events  and  was 
followed  by  a  practice  scenario  including  only  independent  events.  Students  were  also  trained  on 
the  team  dynamics  within  the  simulation,  which  was  also  followed  by  a  practice  scenario 
'  including  both  independent  and  interdependent  events.  In  the  training,  the  number  of  events  as 
well  as  the  complexity  of  informational  content  increased  over  the  two  trainings.  After  each 
training  session,  participants  received  a  computer  generated  team  performance  score  and  had  the 
opportunity  to  review  the  solutions  to  each  event. 

Following  the  training,  students  completed  two  NeoCITIES  performance  scenarios,  each  of 
which  was  14-15  minutes  in  length.  During  each  performance  session,  participants  received  one 
or  two  information  briefings  containing  data  that  was  useful  in  resolving  key  events  in  the 
scenario.  In  addition,  the  NeoCITIES  simulation  was  paused  once  during  each  performance 
scenario  to  allow  participants  to  complete  a  measure  of  situation  awareness.  At  the  end  of  each 
scenario,  students  were  provided  with  computer  generated  team  performance  scores.  Throughout 
the  study,  online  as  well  as  paper  and  pencil  measures  were  collected.  Participants  generally 
found  the  NeoCITIES  simulation  engaging  and  reported  that  it  held  their  interest. 

Each  study  was  preceded  by  extensive  piloting  with  undergraduate  research  assistants  as  well  as 
subject  pool  students.  Piloting  involved  checking  the  technical  features  of  NeoCITIES,  the 
clarity  of  the  training  materials,  the  difficulty  of  scripted  events,  the  timing  of  the  study,  and  the 
variability  in  responses  to  measures.  In  addition,  several  adjustments  were  needed  to  ensure  that 
manipulation  checks  and  measures  were  effective,  and  modifications  were  re-piloted  before 
formal  data  collection  commenced. 

Dependent  Measures 

Dependent  variables  were  information  sharing,  shared  situation  awareness,  team  mental  models, 
and  team  performance.  The  measurement  of  each  will  be  described  briefly  below. 
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Information  sharing  was  measured  qualitatively  via  coding  communication  chat  logs  recorded 
during  the  simulation.  Therefore,  NeoCITIES  allowed  for  a  rich  and  more  objective  indicator  of 
information  sharing  than  self-report  measures.  Undergraduate  research  assistants  were  trained  to 
code  for  information  sharing  on  interdependent  events  regarding  resources  required  (e.g.,  “Police 
should  go  to  Tanker”),  order  (“It’s  Hazmat,  fire,  then  me  for  Tanker”),  pacing  (“You  have  to 
respond  to  the  bank  in  under  45  seconds”),  and  deadlines  (“Ohio  State  fans  defacing  South 
Atherton  at  6:30  today”),  as  well  as  other  categories.  At  least  two  research  assistants  coded  each 
set  of  chat  log  transcripts,  and  interrater  reliability  was  adequate  (typically  above  .7). 

Shared  situation  awareness  refers  to  a  shared  perspective  regarding  current  environmental 
events  (Wellens,  1993).  It  is  comprised  of  three  levels  (Endsley,  1995a).  Level  1  (perception) 
reflects  the  degree  to  which  individuals  perceive  temporal  and  spatial  elements  of  events  in  the 
environment  (e.g.,  a  pilot  noticing  an  image  on  a  radar  screen).  Level  2  (comprehension) 
references  the  extent  to  which  individuals  understand  the  meaning  of  these  events  (e.g., 
understanding  that  the  image  represents  an  approaching  aircraft).  Level  3  (projection)  captures 
the  degree  to  which  individuals  use  these  details  on  events  to  predict  their  status  in  the  near 
future  (e.g.,  planes  will  crash  in  one  minute  if  courses  are  not  corrected;  Endsley,  1995a). 

Shared  situation  awareness  was  assessed  using  three  different  metrics,  one  of  which  was  an 
objective  indicator  (e.g.,  Situation  Awareness  Global  Awareness  Technique;  SAGAT)  and  two 
of  which  were  subjective  indicators  (adaptations  of  the  Mission  Awareness  Rating  Scale 
(MARS;  Matthew  &  Beal,  2002)  and  the  Situational  Awareness  Rating  Technique  (SART, 
Taylor,  1990)).  As  the  most  popular  and  validated  measure  of  situation  awareness  (Endsley, 
1995b),  SAGAT  is  a  freeze  probe  technique  that  requires  the  task  to  be  randomly  stopped  in 
order  to  ask  participants  a  series  of  questions  regarding  their  perceptions  of  the  current  situation. 
NeoCITIES  and  scenario  sequencing  had  to  be  modified  to  integrate  the  SAGAT  freeze  into  the 
simulation  so  that  participants  would  not  be  able  to  predict  and  prepare  for  its  occurrence.  The 
SAGAT  measure  consisted  of  ten  multiple  choice  and  short  answer  questions  based  on 
simulation  events  (e.g.,  Based  on  the  event  description,  what  would  MOST  likely  happen  if  units 
didn’t  arrive  on  scene  within  60  seconds  in  the  case  of  tanker  collision?).  There  was  only  one 
correct  answer  for  each  question  that  could  be  verified  based  on  scenario  scripting  and  a  detailed 
action  history  of  participant  responses. 

In  contrast  to  SAGAT,  which  was  measured  during  NeoCITIES  scenarios,  MARS  and  SART 
were  administered  after  participants  had  played  the  simulation.  In  addition,  whereas  the  SAGAT 
questions  had  to  be  tailored  to  NeoCITIES,  MARS  and  SART  were  generalized  measures  that 
could  be  used  in  a  variety  of  contexts.  Adapted  from  Matthew  and  Beal  (2002),  the  MARS 
measure  was  derived  from  the  most  popular  self-report  measure  of  team  situation  awareness  (the 
Crew  Awareness  Rating  Scale,  CARS;  McGuinness  &  Foy,  2000)  for  use  in  non-military 
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settings.  MARS  consisted  of  five  items  rated  on  a  four-point  behaviorally  anchored  rating  scale. 

A  sample  measure  is,  “Please  indicate  your  team’s  ability  to  identify  changes  in  events  (e.g., 
pacing,  complexity)  throughout  the  scenario.”  On  the  other  hand,  SART  was  adapted  from 
Taylor  (1990),  and  is  considered  to  be  the  most  popular  self-report  measure  of  individual 
situation  awareness  (Salmon,  Stanton,  Walker,  &  Green,  2006).  SART  was  assessed  via  seven 
items  measuring  perceptions  of  elements  in  the  team’s  work  environment  on  a  seven-point  scale. 
A  sample  item  is,  “The  unpredictability  of  the  events  received  in  the  scenario.”  SAG  AT;  MARS, 
and  SART  measures  were  collected  at  two  separate  time  points  within  the  span  of  the 
NeoCITIES  experiments.  < 

Team  mental  models  capture  members’  shared,  organized  understanding  and  mental 
representation  of  relevant  team  knowledge  (Mohammed  &  Dumville,  2001).  Two  properties  of 
team  mental  models  are  sharedness  (the  degree  to  which  members’  mental  models  are  consistent 
with  one  another)  and  accuracy  (the  degree  to  which  members’  mental  models  converge  with 
experts’  mental  models;  Mohammed  &  Hamilton,  2012). 

Three  different  types  of  team  mental  model  content  were  measured:  team  (how  and  with  whom 
work  gets  done),  task  (what  work  gets  done),  and  temporal  (when  work  gets  done).  All  three 
types  were  assessed  using  paired  comparison  ratings,  one  of  the  most  popular  measurement 
techniques  (Mohammed  et  al.,  2010).  Each  mental  model  contained  a  list  of  six  dimensions  that 
were  generated  through  careful  analysis  of  the  task  requirements,  collaborative  processes,  and 
temporal  dynamics  involved  in  NeoCITIES.  The  six  dimensions  were  briefly  defined  and 
arranged  into  a  6  X  5  grid.  Participants  were  asked  to  rate  the  similarity  of  each  dimension  on  a 
scale  of  1  (extremely  unrelated)  to  5  (extremely  related).  Sharedness  was  assed  using  QAP 
correlations  (e.g.,  Mathieu  et  al.,  2000).  Accuracy  was  based  on  the  taskwork,  teamwork,  and 
temporal  team  mental  model  ratings  of  three  subject  matter  experts,  which  correlated  at  .70  or 
higher  before  discussion.  Discrepancies  among  ratings  were  discussed  until  consensus  was 
reached.  The  average  sharedness  of  teams  with  the  expert  ratings  was  assessed  using  QAP 
correlations  (e.g.,  Mathieu  et  al.,  2000). 

In  addition  to  paired  comparison  ratings,  temporal  team  mental  models  were  also  assessed  via 
concept  mapping,  another  common  assessment  technique  used  in  the  literature  (e.g.,  Mohammed 
et  al.,  2012).  After  each  performance  session,  participants  were  given  a  list  of  three  events  and 
asked  who  should  arrive  to  the  event  first,  second,  and  third.  Trained  coders  evaluated  the 
similarity  among  team  member  maps  and  the  number  of  direct  links  shared  in  Studies  1  and  2. 
Agreement  ratings  were  .70  or  higher.  Concept  map  accuracy  was  based  on  the  number  of  roles 
accurately  placed  in  each  sequence.  For  further  information  on  team  mental  model  metrics  within 
NeoCITIES,  refer  to  Mancuso,  Hamilton,  McMillan,  Tesler,  Mohammed,  &  McNeese  (201 1). 
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To  learn  more  about  how  information  sharing,  team  situation  awareness  and  team  mental  models 
were  measured  within  the  NeoCITIES  context,  see  Hamilton  et  al.  2010. 

Team  performance  was  operationalized  objectively  through  the  scores  generated  by  the 
NeoCITIES  simulation.  The  overall  team  performance  score  was  calculated  based  on  an  event 
growth  formula  that  accounted  for  the  number  of  events,  severity  of  events,  and  the  time  taken  to 
successfully  resolve  events.  As  such,  task  performance  in  NeoCITIES  rewards  players  that 
quickly  and  correctly  allocate  the  correct  number  and  type  of  resources  to  an  event.  Players 
receive  lower  scores  when  opportunities  are  lost  through  inaction  or  slow  and/or  incorrect 
responses.  In  addition  to  the  overall  team  performance  score,  NeoCITIES  was  also  modified  to 
provide  more  circumscribed  performance  scores  for  temporally  infused  events,  independent 
events,  and  interdependent  events.  For  more  information  about  NeoCITIES  performance  metrics, 
see  Hellar  (2009)  and  Hellar  and  McNeese  (2010). 

Complexity  of  Data  Collection  and  Analysis 

It  should  be  noted  that  many  of  the  team  cognition  measures  that  we  assessed  are  context- 
dependent  and  therefore  had  to  be  developed  specifically  for  the  NeoCITIES  simulation  (e.g., 
information  sharing  coding;  task,  team,  and  temporal  team  mental  model  paired  comparison 
ratings  and  concept  maps;  SAGAT).  In  addition,  several  parts  of  data  management  and  analysis 
required  coding.  For  example  temporal  mental  models  measured  through  concept  maps  were 
coded  for  the  number  of  direct  links  shared  among  team  members  as  well  as  for  accuracy.  In 
addition,  each  chat  log  was  coded  by  multiple  students  according  to  a  detailed  coding  rubric. 
Determining  the  coding  template,  piloting  the  template,  and  coordinating  the  actual  coding 
required  substantial  time  and  effort. 

For  all  analyses,  the  unit  of  analysis  was  the  team.  The  following  bullet  points  reflect  the 
complexity  of  the  data  analysis: 

•  Complex  coding  required  to  capture  information  sharing  from  chat  logs  across  6 
categories  (measured  twice  in  each  study) 

•  3  types  of  team  mental  models  (taskwork,  teamwork,  temporal) 

•  2  properties  of  team  mental  models  (similarity  &  accuracy) 

•  2  operationalizations  of  team  mental  models 

•  Concept  maps  required  coding  (measured  twice  in  each  study) 

•  Paired  comparison  ratings  translated  into  QAP  correlations 

•  3  operationalizations  of  situation  awareness  (measured  twice  in  each  study) 

•  Various  measures  of  team  performance  (measured  twice  in  each  study) 
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After  briefly  reviewing  the  purpose  and  specific  methodologies  of  the  four  experimental  studies, 
the  results  will  be  presented  below. 


STUDY  1 


Objectives  and  Overview 

The  objectives  of  Study  1  were  to  empirically  investigate  multiple  macrocognitive  processes  in  a 
single  study,  to  infuse  macrocognition  measures  with  a  temporal  focus,  and  to  operationalize  the 
notion  of  a  temporal  team  mental  model.  Specifically,  this  study  manipulated  the  degree  of 
sharing  of  temporal  information  among  participants  (unique  versus  shared)  and  examined  the 
impact  of  this  hidden  profile  task  on  information  sharing,  shared  situational  awareness,  temporal 
mental  models,  and  team  performance.  It  was  expected  that  a  shared  information  profile  would 
positively  impact  these  forms  of  macrocognition. 

In  Study  1 ,  the  following  model  was  tested: 


Figure  5:  Study  1  Model 
Participants  and  Procedure 

The  sample  for  Study  1  consisted  of  216  psychology  undergraduates  (139  females  (66.8%)  and 
69  males  (33.2%)).  Students  were  divided  into  72  three-person  teams  (37  shared  condition  and 
35  unique  condition).  Participants  reported  a  high  level  of  engagement  in  playing  NeoCITIES 
(Mean  =  4.08/5.00).  The  study  took  two  hours  to  complete. 

The  experimental  procedure  consisted  of  the  following  steps: 

•  Online  Survey  1 

•  Basic  NeoCITIES  Training  via  PowerPoint  slides 

•  7  minute  practice  scenario  on  independent  events 

•  Team  Training  via  PowerPoint  slides 

•  6  minute  practice  scenario  on  interdependent  events 

•  Online  Survey  2 

•  Performance  Scenario  1  (14  minutes) 

•  SAGAT  freeze,  1  information  briefing  (shared  or  unique  condition) 


Hidden  Information 
Profile 


•  Information  Sharing 

•  Team  Mental  Models 

•  Team  Situation 
Awareness 


Team 

Performance 
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•  Online  Survey  3 

•  Performance  Scenario  2  (15  minutes) 

•  SAGAT  freeze,  2  information  briefings  (shared  or  unique  condition) 

•  Online  Survey  4  and  Written  Survey 

Information  Sharing  Manipulation 

The  degree  of  information  sharing  among  participants  was  manipulated  through  information 
briefings  delivered  to  participants  at  specified  times  to  coincide  with  events  dispatched 
throughout  the  simulation.  Briefings  contained  temporal  information  about  deadlines  (attack 
expected  at  6:30),  pacing  (all  units  need  to  arrive  on  scene  in  30  seconds  of  simulated  time), 
and/or  ordering  (police  should  arrive  first,  then  fire).  Teams  were  randomly  assigned  to  either  the 
unique  (only  one  team  member  received  the  information)  or  shared  (multiple  team  members 
received  the  information)  condition. 

Results 

As  expected,  the  shared  hidden  profile  condition  resulted  in  increased  overall  information 
sharing  at  Time  1  and  2.  Also  as  predicted,  teams  in  the  shared  information  condition  had  higher 
shared  temporal  team  mental  models  than  those  teams  that  received  unique  information.  These 
effects  occurred  for  both  similarity  ratings  (t  (62)  =  2.40,  p  <  .05)  and  concept  maps  (t  (63)  = 

1 .87,  p  =.067),  but  was  only  marginally  significant  for  the  latter.  These  results  suggest  that  the 
positive  effects  of  open  information  sharing  on  team  performance  (Mesmer-Magnus  & 
DeChurch,  2009)  may  take  place  through  team  mental  model  sharedness.  Furthermore,  the 
shared  information  condition  was  associated  with  a  higher  order  performance  score  at  Time  2. 

Hierarchical  regression  analyses  revealed  that  macrocognitive  processes  had  differential  effects 
on  various  forms  of  team  performance.  As  measured  by  coded  chat  logs,  higher  overall 
information  sharing  resulted  in  higher  team  performance  on  timed  events  at  Time  1  and  2. 

Higher  concept  map  similarity  resulted  in  increased  order  performance  at  Times  1  and  2  and 
increased  overall,  pacing,  and  interdependent  team  performance  at  Time  2.  Higher  teamwork 
similarity  ratings  resulted  in  increased  overall  and  interdependent  team  performance  at  Time  2. 
Higher  temporal  similarity  ratings  increased  the  pacing  score  at  Time  2  (marginal  significance). 
Regarding  situational  awareness,  results  showed  differential  prediction  across  all  three  measures, 
such  that  SART  was  positively  predictive  of  individual  performance,  whereas  team-level 
SAGAT  and  MARS  were  surprisingly  positively  predictive  of  both  individual  and  team 
performance.  Higher  scores  on  MARS  and  SART  were  also  associated  with  higher  engagement 
in  team  processes  and  satisfaction. 


23 


Temporal  team  mental  models  were  not  significantly  correlated  with  teamwork  or  taskwork  team 
mental  models.  The  relationships  between  temporal  team  mental  models  measured  through 
concept  maps  and  those  measured  through  paired  comparison  ratings  had  distinct  effects  on  team 
performance  for  temporal  events.  Temporal  team  mental  models  had  a  positive,  significant 
relationship  with  team  performance  when  assessed  via  concept  maps,  but  not  when  assessed  via 
similarity  ratings.  Consistent  with  these  results,  a  recent  meta-analysis  concluded  that  the  effects 
of  team  mental  models  on  performance  vary  according  to  the  type  of  measure  used  to  evaluate 
the  construct  (DeChurch  &  Mesmer-Magnus,  2010b). 

Conclusions 

As  hypothesized,  a  shared  information  profile  positively  impacted  various  forms  of 
macrocognition.  Moreover,  results  revealed  that  team  cognition  matters  for  team  performance. 
Several  types  of  team  cognition  were  positively  related  to  team  performance,  but  there  was  a 
differential  pattern  of  results  in  that  information  sharing  and  temporal  mental  models  were  more 
positively  related  to  timing  performance  metrics  whereas  shared  situation  awareness  was  more 
positively  related  to  overall  team  performance  and  a  broader  range  of  performance  indicators. 
These  results  contribute  to  the  team  literature  because  understanding  the  differential  effects  of 
multiple  forms  of  team  cognition  has  been  identified  as  a  key  research  need  (e.g.,  Mohammed  et 
al.,  2010;  Salas  &  Wildman,  2009).  In  addition,  team  cognition  assessment  methods  have  been 
temporally  deficient  (Mohammed  et  al.,  2012). 

Temporal  mental  models  were  found  to  show  a  distinct  pattern  of  results  from  more  traditional 
team  mental  model  content.  Specifically,  temporal  mental  models  exhibited  a  different  pattern 
with  team  performance  than  teamwork  mental  models.  High  performing  teams  had  both  high 
sharedness  and  accuracy  of  temporal  mental  models,  but  either  high  sharedness  or  accuracy  of 
teamwork  mental  models.  Future  work  should  further  examine  the  role  of  temporal  mental 
models  as  a  new  content  domain  given  that  it  was  non-redundant  with  more  traditional  forms  of 
mental  models  in  this  study.  As  failure  to  agree  on  how  temporal  resources  should  be  allocated 
among  team  members  can  seriously  jeopardize  team  outcomes,  the  conceptualization  and 
operationalization  of  a  temporal  team  mental  model  demonstrates  promise  for  future  research. 

STUDY  2 

<? 

Objectives  and  Research  Questions 

The  objectives  of  Study  2  were  the  conceptualization  and  operationalization  of  storytelling  as  a 
complex  form  of  metaphor  and  to  explore  the  use  of  storytelling  as  a  team  training  tool. 
Specifically,  we  examined  the  influence  of  four  types  of  stories  and  a  control  group  (no  story)  on 
information  sharing,  team  mental  models,  and  team  performance.  In  terms  of  research  questions, 
we  explored  whether  storytelling,  as  a  team  training  intervention,  improves  information  sharing, 
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team  mental  models,  situation  awareness,  and  team  performance.  In  addition,  we  investigated 
whether  different  types  of  storytelling  differentially  affect  outcomes.  For  example,  collaboration 
stories  may  be  more  likely  to  influence  teamwork  mental  model  sharedness  and  accuracy, 
whereas  timing  stories  would  be  more  likely  to  influence  temporal  mental  model  sharedness  and 
accuracy. 

Story  Development 

We  first  conducted  a  review  of  the  literature  on  metaphors.  Given  the  disconnect  between  the 
types  of  metaphors  discussed  in  the  literature  and  what  would  be  relevant  in  a  simulated 
emergency  management  context,  we  turned  to  storytelling,  a  more  complex  version  of 
metaphors.  We  conducted  a  literature  review  on  the  storytelling  literature  and  consulted  with 
Steve  Fiore,  who  provided  us  with  several  of  his  relevant  papers  on  the  intersection  of  narrative 
and  distributed  teams.  Although  Fiore’s  work  provided  us  with  an  excellent  conceptual 
framework  for  integrating  storytelling  and  teams,  the  lack  of  empirical  work  in  this  area  meant 
that  we  had  little  guidance  in  writing  stories  that  would  work  in  a  team  simulation  context  and 
developing  measures  to  test  their  effectiveness.  Our  research  team  spent  many  hours 
brainstorming  to  come  up  with  ideas  for  the  content  of  the  stories  (sports  context,  design  teams) 
as  well  as  discussing  the  logistics  of  the  story  medium  (e.g.,  written,  auditory,  illustrated).  It  took 
several  months  of  planning  before  we  decided,  after  much  discussion  and  deliberation,  on  the 
four  stories  described  below. 

Story  Content:  Collaboration  and  Timing.  Metaphors  play  an  important  role  in  surfacing 
tensions  and  ambiguities  inherent  in  constructs.  As  metaphors  are  especially  valuable  when 
exploring  complex,  novel,  and  poorly  understood  phenomena,  we  identified  areas  of  the 
experiment  and  the  NeoCITIES  interface  where  students  commonly  experienced  confusion  and 
were  in  need  of  further  instruction.  Based  on  observations  and  qualitative,  self-report  data  from 
Study  1,  two  participant  problem  areas  important  to  success  in  the  NeoCITIES  simulation  were 
virtual  collaboration  across  units  (police,  fire,  hazmat)  and  temporal  sequencing  (which  units 
should  arrive  first,  second,  and  third).  Therefore,  we  directed  our  efforts  toward  story  creation  in 
these  areas  that  were  relevant  within  an  emergency  management  context.  The  collaboration 
theme  highlighted  the  breakdowns  that  occur  when  teams  fail  to  communicate  critical 
information,  and  the  timing  theme  highlighted  the  breakdowns  that  occur  when  teams  fail  to 
attend  to  the  pacing  and  ordering  of  member  actions 

Story  Context:  Metaphorical  versus  Analog  Stories.  In  addition  to  the  story  content,  decisions 
had  to  be  made  concerning  story  context.  One  view  is  that  sensemaking  is  enhanced  by 
comparing  new  concepts  that  have  not  been  directly  or  indirectly  experienced  to  what  is  already 
known.  In  this  line  of  thinking,  if  a  metaphor’s  content  is  familiar  to  the  listener  and  couched  in  a 
story  to  increase  listener  engagement,  then  any  metaphor  presented  within  the  framework  of  a 
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narrative  could  potentially  invoke  learning  (e.g.,  Gilman  &  Gillan,  2001).  Therefore  the  term 
“metaphorical  storytelling”  is  used  to  refer  to  stories  that  help  to  connect  what  listeners  are  less 
familiar  with  to  what  they  are  more  familiar  with.  Since  stories  should  be  relatable  (Denning, 
2001)  and  popular  TV  shows  have  popularized  this  setting,  the  metaphorical  story  occurred  in  a 
medical  context.  The  medical/metaphorical  story  was  based  on  a  true  story  that  occurred  at  a 
hospital  in  New  York  City.  In  the  story,  the  protagonist  breaks  his  leg  during  a  winter  vacation. 
The  patient’s  post-surgery  recovery  encountered  complications  that  stemmed  from  collaboration 
and  timing  errors  of  the  medical  team. 

In  contrast  to  metaphorical  storytelling,  analog  storytelling  maintains  a  closer  connection 
between  the  story  and  the  target  domain.  Derived  from  analogical  problem  solving  (Bransford  & 
Stein,  1984),  the  goal  of  analog  storytelling  is  to  apply'the  principles  of  narrative  to  a  domain  to 
a  similar  situation  that  the  story  represents.  The  NeoCITIES/analog  story  focused  on  a  graduate 
student  working  in  a  laboratory  environment  during  a  severe  snowstorm.  During  an  unforeseen 
power  outage,  the  protagonist  has  an  accident  where  a  beaker  of  acid  he  was  carrying  splashed 
its  contents  on  his  left  arm.  The  character  has  severe  chemical  bums  and  requires  the  assistance 
of  police,  fire,  and  hazmat.  Complications  stem  from  collaboration  and  timing  errors  of  the 
response  team. 

Analog  storytelling  may  be  more  effective  than  metaphorical  storytelling  in  that  the  conceptual 
leap  between  the  story  and  the  target  domain  is  smaller,  allowing  participants  to  more  easily  see 
the  connections.  In  contrast,  metaphorical  storytelling  may  prove  superior  to  analog  storytelling 
because  of  its  greater  versatility  and  its  ability  to  link  a  domain  that  is  more  familiar  with  a 
domain  that  is  less  familiar  to  participants. 

Deep  versus  surface-level  structure .  Before  writing  the  stories,  we  first  had  to  identify  the  deep 
structural  elements  of  NeoCITIES  to  ensure  that  the  stories  had  parallels  to  the  simulation  that 
could  improve  the  user’s  performance.  NeoCITIES  subject  matter  experts  listed  several  deep 
structure  elements,  including  three  distinct  roles,  team  monitor,  overt  communication  between 
roles,  and  sequencing.  With  the  medical/metaphorical  story,  the  deep  level  structure  was  similar 
to  NeoCITIES,  but  the  surface  level  structure  was  distinct.  On  the  other  hand,  both  surface  and 
deep  structure  were  similar  to  the  simulation  for  the  NeoCITIES/analog  story. 

The  process  of  story  construction.  Our  stories  were  designed  to  embed  the  deep  structure  about 
the  NeoCITIES  simulation  into  an  engaging  and  applicable  story  format  using  the  principles  of 
narrative.  Each  of  the  four  stories  went  through  several  drafts,  informed  by  internal  discussions 
among  the  research  team  as  well  as  sources  with  medical  expertise  (for  the  medical  stories)  and 
emergency  crisis  management  experience  (for  the  NeoCITIES  stories).  We  had  to  balance  the 
need  to  incorporate  the  deep  level  structure  of  collaboration  and  timing  requirements  in  the 
NeoCITIES  simulation  with  the  need  for  the  story  to  be  realistic,  credible,  and  engaging.  We 
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consulted  with  a  professor  and  writer  from  the  English  department  to  edit  our  stories  and  ensure 
that  they  met  the  principles  of  narrative. 

Several  pilot  studies  were  conducted  to  refine  the  stories  and  test  participants’  reactions.  After 
conducting  informal  pilots  with  undergraduate  research  assistants,  we  conducted  a  formal  pilot 
study  with  participants  from  the  subject  pool.  Based  on  the  valuable  feedback  provided  from  all 
of  these  sources,  we  successively  revised  the  stories  as  well  as  the  power  point  illustrations.  In 
order  to  introduce  more  experimental  control,  we  felt  that  it  was  important  for  the  stories  to  be 
audio-recorded  and  accompanied  by  pictures  illustrating  the  major  points  on  PowerPoint  slides 
that  were  synchronized  to  the  audio  recording.  In  piloting,  students  reacted  positively  to  this 
storytelling  format.  Thus,  with  successive  edits,  the  stories  had  to  be  re-recorded  and  timed  with 
power  point  slides  many  times  throughout  the  course  of  the  semester.  In  addition  to  participant 
feedback  with  regard  to  realism,  credibility,  interest  level,  and  affect,  we  also  needed  to  ensure 
that  the  four  stories  were  similar  in  length  and  structure.  For  more  information  about  how  the 
stories  were  constructed,  see  Mancuso,  Parr  et  al.  (201 1). 

Storytelling  Manipulation 

Four  types  of  stories  (medical  timing,  medical  collaboration,  NeoCITIES  timing,  and 
NeoCITIES  collaboration)  and  a  control  group  (no  story)  were  constructed.  The  four  stories 
allowed  us  to  test  the  effect  of  story  content  (timing  versus  collaboration)  as  well  as  story  context 
(medical  versus  emergency  crisis  management)  on  macrocognitive  processes.  In  the  medical 
context,  the  deep  level  structure  was  similar  to  NeoCITIES,  but  the  surface  level  structure  was 
distinct  (metaphorical  storytelling).  In  contrast,  in  the  NeoCITIES  context,  the  surface  and  deep 
level  structures  were  similar  (analog  storytelling).  Each  story  was  approximately  four  and  a  half 
minutes  in  length. 

In  the  control  condition,  participants  were  given  a  description  of  a  college  EMS  service  in  a 
factual  form  that  did  not  meet  the  requirements  of  a  story.  The  description  was  written  to  be  of 
similar  length  as  the  four  stories  and  had  a  similar  surface  structure  (emergency  response)  to 
NeoCITIES.  However,  unlike  the  stories,  it  did  not  have  deep  structure  nor  incorporate  the 
principles  of  narrative. 

Participants 

Participants  consisted  of  294  individuals  (52%  Male,  47.1%  female)  divided  into  98  three-person 
teams.  The  sample  was  78.2  percent  Caucasian,  52.9  percent  male,  and  41.0  percent  junior 
students.  The  average  age  of  the  sample  was  20.54  (SD  =  1 .39).  On  a  scale  of  1  (none)  to  5 
(extensive),  participants  reported  having  little  to  moderate  levels  of  experience  working  in  a 
virtual  team  (mean  =  2.23,  SD  ~  0.81). 
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The  five  story  conditions  were  roughly  equal  in  terms  of  team  sample  size: 

•  4  story  conditions: 

•  Medical  Collaboration  (n=18) 

•  Medical  Timing  (n=19) 

•  NeoCITIES  Collaboration  (n=20) 

•  NeoCITIES  Timing  (n=  19) 

•  Control  condition  (n=22) 

Experimental  Procedure 

The  experimental  procedure  consisted  of  the  following  steps: 

•  Basic  NeoCITIES  Training  Video 

•  5  minute  practice  scenario  on  independent  events 

•  Team  Training  Video 

•  Storytelling  Video  (4.5  minutes) 

•  Online  manipulation  check 

•  5  minute  practice  scenario  on  interdependent  events 

•  Performance  Scenario  1  (14  minutes) 

•  SAGAT  freeze,  1  information  briefing 

•  Online  Survey 

•  Performance  Scenario  2  (15  minutes) 

•  SAGAT  freeze,  2  information  briefings 

•  Written  and  online  surveys 

In  addition  to  the  storytelling  manipulation,  Study  2  differed  from  Study  1  in  that  only  unique 
information  briefings  were  distributed  (only  one  participant  received  each  piece  of  information). 
In  addition,  the  experiment  schedule  had  to  be  lengthened  from  2  to  2  and  Vi  hours.  We  also 
moved  to  video-based  training,  which  was  an  improvement  over  the  PowerPoint  slides  used  in 
Study  1 .  Thus,  we  introduced  increased  technical  demands  with  the  storytelling  study  beyond  the 
challenges  associated  with  running  the  basic  NeoCITIES  simulation. 

Results 

The  storytelling  manipulation  checks  were  generally  effective.  Participants  in  the  collaboration 
conditions  reported  that  the  stories  reflected  more  communication  than  timing.  Participants  in  the 
timing  conditions  reported  that  the  stories  reflected  both  timing  and  collaboration.  Given  that 
sequencing  encompassed  both  collaboration  and  timing,  the  latter  finding  is  not  unexpected.  In 
comparison  to,  control  condition  participants,  students  receiving  the  metaphor  and  analog  story 
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conditions  indicated  that  the  story  was  more  interesting,  understandable,  engaging,  surprising, 
and  more  connected  to  NeoCITIES. 

Surprisingly,  storytelling  did  not  significantly  influence  information  sharing,  team  mental 
models,  situation  awareness,  or  team  performance.  Differences  in  story  types  were  not  salient  to 
participants.  One  of  the  reasons  why  storytelling  did  not  exhibit  direct  effects  on  metacognition 
and  performance  may  have  been  due  to  the  timing  of  storytelling.  Introduced  as  part  of  training, 
participants  may  have  been  overwhelmed  with  new  information,  attenuating  the  effects  of  the 
story.  In  Study  3,  we  introduced  stories  after  the  team  had  become  more  familiar  with 
NeoCITIES.  Another  reason  why  storytelling  did  not  have  more  of  an  impact  in  this  study  may 
have  been  that  participants  needed  more  opportunity  to  reflect  on  the  stories  for  them  to 
influence  behavior.  In  Study  3,  we  explore  the  interactive  effects  of  storytelling  and  reflexivity 
(intervention  to  allow  teams  to  overtly  think  about  their  performance  and  goals). 

\ 

Consistent  with  the  results  from  Study  1 ,  team  cognition  was  found  to  positively  impact  team 
performance,  but  differential  patterns  were  found  depending  on  specific  outcomes.  For  example, 
shared  temporal  team  mental  models  had  a  positive  effect  on  team  performance  (percentage  of 
interdependent  events  completed  correctly)  when  measured  as  both  concept  maps  (p  =  0.26,/?  < 
.01)  and  paired  comparison  ratings  (p  =  0.28,  p  <  .01).  In  addition,  temporal  team  mental  models 
operationalized  as  concept  maps  and  paired  comparison  ratings  accounted  for  an  additional  13% 
of  the  variance  in  team  performance  (percentage  of  interdependent  events  completed  correctly) 
beyond  teamwork  and  taskwork  team  mental  models  and  controls  (F  2, 86  =  7.41,  p  <  .01). 

Hierarchical  regression  analyses  revealed  a  significant  interaction  between  shared  teamwork  and 
shared  temporal  team  mental  models  on  overall  duration  (how  long  it  took  team  members  to 
respond  to  events)  after  controls  (story  conditions,  performance  at  Time  1)  and  teamwork, 
taskwork,  and  temporal  team  mental  model  sharedness  main  effects  (p  =  -0.22,/?  <  .05;  AR2=.05, 
p<.10).  As  shown  in  Figure  6,  faster  responding  teams  were  on  the  same  page  regarding  both 
taskwork  and  temporal  elements  of  the  task.  High  taskwork  sharedness  without  temporal 
sharedness  was  associated  with  slower  response  times. 
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Figure  6:  Interaction  between  Taskwork  and  Temporal  Team  Mental  Model  Sharedness  on 
Overall  Team  Performance 

Hierarchical  regression  analyses  also  revealed  a  significant  interaction  between  concept  map  and 
paired  comparison  rating  measures  of  temporal  team  mental  models  on  overall  team  performance 
(P  =  <  .05;  AR2=.025,  p<.05).  As  shown  in  Figure  7,  being  low  on  both  temporal  team 

mental  model  measures  lowers  performance,  but  being  high  on  both  does  not  advantage  teams. 
Rather,  scoring  high  on  one  measure  compensates  for  scoring  low  on  the  other  measure. 
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Figure  7:  Interaction  between  Temporal  Team  Mental  Model  Sharedness  Measured  via  Concept 
Maps  and  via  Paired  Comparison  Ratings  on  Team  Performance 

Conclusions 

Because  anecdotal  and  conceptual  work  has  dominated  the  storytelling  literature,  Study  2 
focused  on  the  operationalization  of  storytelling  (as  a  complex  form  of  metaphor)  in  team 
cognition  research.  Although  storytelling  has  been  proposed  to  be  effective  in  distributed  teams, 
but  not  empirically  verified,  we  explored  the  use  of  storytelling  as  a  team  training  tool.  Contrary 
to  predictions,  results  revealed  that  storytelling  did  not  directly  affect  team  cognition  or 
performance.  However,  several  future  research  directions  were  outlined  in  preparation  for  Study 
3. 

As  in  Study  1 ,  we  examined  team  mental  models  through  a  more  sophisticated  lens  than  previous 
studies  in  that  taskwork,  teamwork,  and  temporal  team  mental  models  were  examined  and  both 
concept  mapping  and  paired  comparison  ratings  were  measured  for  temporal  team  mental 
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models.  Showing  promise  for  future  research,  temporal  mental  models  were  distinct  from 
taskwork  and  teamwork  mental  models  and  added  unique  variance  beyond  these  traditional 
measures.  The  impact  of  team  mental  models  on  performance  was  dependent  on  the  type  of 
dependent  variable  and  operationalization  assessed. 

STUDY  3 


Purpose  &  Overview 

The  purpose  of  the  third  study  was  to  examine  storytelling  and  guided  reflexivity  as  two 
potential  contributors  to  macrocognitive  processes  and  team  performance.  Guided  reflexivity  is  a 
planned  intervention  to  allow  teams  to  overtly  reflect  on  their  performance  and  goals.  We 
furthered  empirical  examination  of  storytelling  in  teams  by  examining  its  impact  after  teams  had 
gained  some  experience  with  the  NeoCITIES  simulation  (as  opposed  to  during  team  training  as 
in  Study  2).  Some  researchers  have  suggested  that  stories  can  be  beneficial  at  any  time  (Fiore, 
Johnston,  &  McDaniel,  2007),  while  others  have  proposed  that  in  general,  interventions  should 
be  made  after  a  team  has  gained  some  experience  with  the  task  at  hand  (Gurtner,  Tschan, 
Semmer,  &  Nagele,  2007;  Hackman  &  Wageman,  2005). 

Below  is  the  model  examined  in  Study  3: 


Figure  8:  Study  3  Model 
Participants 

Participants  included  321  undergraduate  students  comprising  107  three-person  teams.  The 
sample  was  74.1  percent  Caucasian  and  48.6  percent  male.  The  average  age  of  the  sample  was 
20.62  (SD  =  2.33).  On  a  scale  of  1  (none)  to  5  (extensive),  participants  reported  having  little  to 
moderate  levels  of  experience  working  in  a  virtual  team  (mean  =  2.23,  SD  =  1.14). 
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Experimental  Design 


The  study  design  was  a  fully-crossed  2  (storytelling)  x  2  (guided  team  reflexivity)  factorial 
design  (see  Figure  9  below).  The  study  duration  was  2.5  hours. 


* 
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None  (Control) 

Storytelling 
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X 

X 

None  (Control) 

X 

X 

Figure  9:  The  Experimental  Design  for  Study  3 

Storytelling  Manipulation 

Participants  viewed  a  four  and  a  half  minute  video  that  told  the  story  of  a  graduate  student  who 
incurred  serious  health  injuries  due  to  a  lack  of  coordination  and  timing  by  his  emergency 
response  team.  A  narrator’s  voice  was  heard  while  PowerPoint  slides  showing  pictures  and  other 
visual  summaries  of  the  key  points  of  the  story  were  displayed.  The  objective  of  the  story  was  to 
convey  the  message  that  team  members  needed  to  collaborate  with  each  other  and  time  their 
responses  well  (i.e.,  meet  deadlines,  arrive  at  events  in  the  proper  order)  if  they  were  to  succeed 
in  their  scenario.  For  the  storytelling  control  groups,  the  participants  viewed  a  video  describing 
the  same  objectives  of  collaboration  and  timing,  but  in  a  non-storytelling  format.  In  order  to 
equalize  the  time  with  the  storytelling  conditions,  participants  completed  a  3.5  minute  filler 
survey  and  listened  to  a  one  minute  video  explaining  the  importance  of  sequencing  and  meeting 
deadlines,  which  was  the  same  as  the  deep  structure  of  the  story  condition.  In  this  way,  we 
contrasted  the  value  of  a  story  with  just  communicating  the  base  information. 

In  contrast  to  Study  2,  the  storytelling  manipulation  was  administered  after  Scenario  1  instead  of 
as  part  of  training.  In  addition,  only  the  NeoCITIES  timing  story  was  selected  from  the  four 
stories  contrasted  in  Study  2.  Also  in  contrast  to  the  control  condition  in  Study  2  (similar  surface 
structure,  but  no  deep  structure),  we  communicated  the  same  deep  structure  as  the  NeoCITIES 
simulation. 

Reflexivity  Manipulation 

After  the  storytelling  intervention,  participants  in  the  guided  team  reflexivity  condition  were 
given  six  minutes  (based  on  extensive  pilot  testing)  to  reflect  on  their  performance  via  chat. 

Three  questions  were  used  to  guide  the  discussion,  including,  “What  were  the  main  points  you 
learned  from  the  story  that  your  team  can  utilize  while  playing  NeoCITIES?”,  “What  went  right 
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and  wrong  in  the  way  your  team  performed  in  Scenario  1?”,  and  “Please  come  up  with  strategies 
to  improve  your  performance.”  In  the  control  group,  the  participants  were  told  to  discuss 
whether  technology  was  a  benefit  or  detriment  to  society  and  interpersonal  relationships  for  an 
identical  amount  of  time.  Piloting  revealed  that  this  topic  was  face  valid  in  tenns  of  relevance  to 
NeoCITIES,  but  unrelated  to  strategizing  in  the  simulation. 

Experimental  Procedure 

The  addition  of  the  reflexivity  condition  necessitated  increasing  the  length  of  Study  3  to  2  and  l/2 
hours.  The  experimental  procedure  occurred  in  the  following  steps: 

•  Basic  NeoCITIES  Training  Video  &  Practice 

•  Team  Training  Video  &  Practice 

•  Performance  Scenario  1(14  minutes) 

•  SAGAT  freeze,  1  information  briefing 

•  Online  survey 

•  Storytelling  video  or  control  (4.5  minutes) 

•  Online  manipulation  check 

•  Reflexivity  manipulation  or  control 

•  Online  manipulation  check 

•  Performance  Scenario  2  (15  minutes) 

•  SAGAT  freeze,  2  information  briefings 

•  Written  and  online  surveys 

Results 

Storytelling  and  reflexivity  manipulations  were  effective.  All  of  the  hypotheses  were  tested  at  the 
team-level  using  hierarchical  regression.  We  controlled  for  video  game  experience  as  well  as 
knowledge  of,  and  experience  in,  emergency  response  and  hospital  settings.  Specifically,  we 
wanted  to  ensure  that  team  outcomes  were  not  due  to  experienced  gamers  learning  the  simulation 
more  quickly,  or  an  unfair  advantage  for  participants  with  greater  insight  into  how  emergency 
events  should  be  solved. 

Storytelling  was  found  to  have  a  positive  effect  on  team  mental  model  similarity  (P=  .251,  p= 
.011).  That  is,  team  members  who  learned  the  importance  of  collaboration  and  timing  for 
successfully  playing  NeoCITIES  via  a  story  had  more  similar  views  on  how  to  solve  events  than 
team  members  who  were  taught  the  same  lessons  through  a  more  straightforward,  non-story 
approach. 

Guided  team  reflexivity  did  not  have  a  significant  relationship  with  team  mental  model  similarity 
(P=  -.004,  ns).  That  is,  simply  being  given  the  opportunity  to  discuss  their  performance  and 
strategize  for  the  future  did  not  improve  teammates5  team  mental  model  similarity.  However,  the 
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interaction  of  storytelling  and  reflexivity  predicted  variance  in  team  mental  model  similarity 
above  and  beyond  storytelling  alone  (p=  .197,  p=.043).  As  seen  in  Figure  10,  teams  who 
received  a  story  and  had  an  opportunity  to  reflect  on  their  performance  performed  the  best,  while 
those  who  only  received  one  or  the  other  (or  neither)  performed  worse.  In  other  words, 
storytelling  may  provide  the  learning  content  but  reflexivity  provides  the  mechanism  by.  which 
the  content  can  be  fully  processed.  Having  a  discussion  to  create  strategies  and  “get  on  the  same 
page”  may  not  be  useful  if  teammates  do  not  have  common  knowledge  on  which  to  base  their 
discussion. 


Interactive  Effect  of  Storytelling  and 
Reflexivity  on  Concept  Map  Similarity 


Reflexivity  Condition 


Story  Condition 

- none 

- yes 


Figure  1 0:  The  interactive  effect  of  storytelling  and  reflexivity  on  team  mental  model  similarity 
(concept  mapping). 

Conclusions 


We  contributed  to  the  literature  by  examining  two  unexplored  antecedents  of  team  mental 
models,  which  has  been  identified  as  a  research  need  in  a  recent  review  article  (Mohammed  et 
al.,  2010).  In  addition,  there  has  been  little  empirical  examination  of  reflexivity  as  a  planned 
team  intervention.  Results  revealed  that  in  the  absence  of  media-rich  communication 
opportunities,  the  combination  of  presenting  important  information  in  story  format  and  giving 
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team  members  time  to  reflect  upon  their  strategies  improved  team  mental  model  similarity, 
which  in  turn  improved  team  performance.  For  more  information  about  the  design,  results,  and 
conclusions  of  Study  3,  see  Tesler  (2011)  and  Tesler  et  al.  (201 1). 

STUDY  4 


Purpose  and  Hypotheses 

As  shown  in  Study  3,  the  positive  effects  of  stories  and  guided  team  reflexivity  on  team  mental 
model  similarity  are  enhanced  when  used  in  tandem  (Tesler  et  al.,  2011).  Expanding  upon  Study 
3,  the  purpose  of  Study  4  was  to  further  examine  the  conditions  under  which  storytelling  would 
influence  team  cognition  and  team  performance.  Specifically,  we  expanded  the  reflexivity 
manipulation  to  compare  and  contrast  individual  storytelling  reflexivity  and  group  reflexivity 
conditions.  Thus,  we  investigated  the  combined  effects  of  three  interventions-  storytelling,  and 
guided  individual  and  team  reflexivity-  on  team  mental  model  similarity  and  performance  in 
distributed  teams. 

Guided  individual  reflexivity  (GIR)  occurs  when  an  intervention  calls  for  individual  members  to 
reflect  on  their  performance  and  how  they  can  improve.  Guided  team  reflexivity  (GTR)  occurs 
when  the  intervention  is  at  the  group-level,  requiring  all  team  members  to  collectively  reflect  on 
their  performance  and  develop  strategies  for  improvement.  Although  individual  reflection  may 
increase  one’s  understanding  of  what  needs  to  be  done  and  how,  there  is  no  guarantee  that  all 
team  members  will  derive  the  same  conclusions.  Thus,  we  hypothesized  that  GTR  would  be 
more  effective  in  increasing  team  mental  model  similarity  than  GIR. 

Having  the  opportunity  to  individually  reflect  on  a  story’s  meaning  should  allow  team  members 
to  more  efficiently  use  their  knowledge  to  jointly  create  strategies  for  improving  future 
performance.  Primed  to  consider  how  the  lessons  from  the  story  could  apply  to  their 
performance,  individual  reflexivity  participants  should  more  quickly  transition  the  group  effort  to 
creating  specific  performance  strategies.  However,  when  the  same  message  is  presented  in  non¬ 
story  format,  individual  reflection  should  be  less  needed  due  to  the  message’s  straightforward 
nature.  Thus,  we  hypothesized  that  the  combined  presence  of  GIR  and  GTR  would  result  in  more 
similar  team  mental  models  than  GTR  alone  when  a  storytelling  intervention  was  also  present. 
We  further  proposed  that  team  mental  model  similarity  would  mediate  the  relationship  between 
said  reflexivity-storytelling  interaction  and  team  performance. 

Participants  &  Design 

Data  was  collected  from  185  three-person  undergraduate  teams  at  a  large  Mid-Atlantic 
university.  The  sample  .was  68.8  percent  Caucasian  and  41.4  percent  male.  The  average  age  of 
the  sample  was  20.13  (SD  =  1.94).  On  a  scale  of  1  (none)  to  5  (extensive),  participants  reported 
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having  little  to  moderate  levels  of  experience  working  in  a  virtual  team  (mean  =  1.90,  SD  = 
1.03). 

Data  was  collected  on  a  six-cell  experimental  design  (storytelling/control;  individual  storytelling 
reflexivity/control;  group  reflexivity/control). 

The  study  design  was  a  2  (storytelling)  x  3  (guided  individual  reflexivity,  guided  team 
reflexivity,  or  both)  design  with  6  conditions  (see  Figure  1 1).  Because  we  were  only  interested 
in  the  relative  contributions  of  GIR  and  GTR  and  the  effects  of  reflexivity  versus  no  reflexivity 
have  been  supported  in  prior  studies  (Tesler,  201 1;  Tesler  et  al.,  201 1),  we  did  not  include 
conditions  where  neither  was  present.  The  study  duration  was  2.5  hours. 


Guided  Reflexivity 

Individual' Team  Both 

Present 

X 

X  X 

Storytelling 

None  (Control) 

X 

X  X 

Figure  1 1 :  The  experiment  design,  with  six  conditions. 


Storytelling  Manipulation 

Participants  viewed  the  same  five  minute  video  as  in  Study  3  that  told  the  story  of  a  graduate 
student  who  incurred  serious  health  injuries  due  to  a  lack  of  coordination  and  timing  by  his 
emergency  response  team.  A  narrator’s  voice  was  heard  while  PowerPoint  slides  showed 
pictures  and  other  visual  summaries  of  the  key  points.  The  objective  of  the  story  was  to  convey 
the  message  that  team  members  needed  to  collaborate  with  each  other  and  time  their  responses 
well  (i.e.;  meet  deadlines,  arrive  at  events  in  the  proper  order)  to  succeed  in  their  scenarios.  For 
the  storytelling  control  groups,  participants  viewed  a  video  describing  the  same  objectives  of 
collaboration  and  timing,  but  in  a  non-storytelling  format. 

GIR  Manipulation 

After  the  storytelling  intervention,  participants  in  the  GIR  condition  were  given  an  exercise 
involving  matching  items  from  one  column  (i.e.,  important  elements  from  the  storytelling/control 
video),  to  their  correct  counterparts  in  the  second  column  (the  meanings  of  those  elements  as 
they  related  to  successful  performance  in  NeoCITIES).  For  example,  the  item  “Police  had  to 
clear  the  roads  so  that  Hazmat  could  identify  the  chemical  and  then  Fire  could  treat  Dan”  would 
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be  matched  to  “The  order  in  which  resources  arrive  is  often  important.”  Thus,  participants  were 
given  a  structured  opportunity  to  make  the  connections  between  the  story  and  how  it  could  help 
improve  performance  in  NeoCITIES.  In  the  control  condition,  participants  were  not  given  any 
matching  task. 

GTR  Manipulation 

Next,  participants  in  the  GTR  condition  were  given  six  minutes  (based  on  extensive  pilot  testing) 
to  reflect  on  their  performance  via  chat.  Three  questions  were  used  to  guide  the  discussion, 
including,  “How  well  do  you  think  you  and  your  team  just  performed  in  Performance  Scenario 
1?”  and  “Please  come  up  with  strategies  to  improve  your  performance.”  In  the  control  group, 
participants  were  told  to  discuss  an  unrelated  topic  on  technology  and  social  relationships  for  an 
identical  amount  of  time. 

Experimental  Procedure 

The  experimental  procedure  occurred  in  the  following  steps: 

•  Basic  NeoCITIES  Training  Video  &  Practice 

•  Team  Training  Video  &  Practice 

•  Performance  Scenario  1  (14  minutes) 

•  SAGAT  freeze,  1  information  briefing 

•  Online  survey 

•  Storytelling  video  or  control  (4.5  minutes) 

•  Online  manipulation  check 

•  Individual  Reflexivity  manipulation  or  control 

•  Online  manipulation  check 

•  Group  Reflexivity  manipulation  or  control 

•  Online  manipulation  check 

•  Performance  Scenario  2  (15  minutes) 

•  SAGAT  freeze,  1  information  briefing 

•  Written  and  online  surveys 


Results 

Storytelling  and  reflexivity  manipulations  were  effective.  All  of  the  hypotheses  were  tested  at 
the  team-level  using  hierarchical  regression.  We  controlled  for  video  game  experience  and 
knowledge  and  experience  in  emergency  response  and  hospital  protocols  in  addition  to  gender 
composition  (i.e.,  percentage  of  females  in  a  team). 

Testing  the  effect  of  GTR  versus  GIR  on  team  mental  model  similarity,  we  found  that  GTR 
indeed  had  a  significantly  larger  positive  effect  than  GIR  on  concept  map  similarity  (P=  .21,  p< 
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.05).  Therefore,  if  faced  with  the  choice,  a  team-level  reflexivity  intervention  will  produce 
members  with  more  similar  views  on  how  events  should  be  solved. 

When  testing  whether  the  positive  effects  of  GTR  on  team  mental  model  similarity  are  increased 
by  adding  GIR,  we  found  a  significant  interaction  with  storytelling  ((3=  -.26,  p<  .01;  Figure  12). 
Interestingly,  simple  slope  analyses  revealed  that  GTR  alone  was  significantly  better  at 
increasing  taskwork  mental  model  similarity  than  GTR  and  GIR  combined  in  the  storytelling 
condition  (p=  -.28,  p<  .05).  In  fact,  storytelling  plus  GTR  had  the  best  outcome  overall  (J3=  .16, 
p<  .05).  As  expected  there  was  no  significant  difference  in  the  no-story  condition  (p=  .24,  ns ), 


GTR  G1R+GTR 


Figure  12:  The  interactive  effect  of  storytelling  and  reflexivity  on  team  mental  model  similarity 
(similarity  ratings). 

As  expected,  we  found  a  significant  positive  relationship  between  team  mental  model  similarity 
and  team  performance  for  both  concept  maps  (p-  .16,  p<  .05)  and  paired  comparison  ratings  (p= 
.15,  p<  .05).  Utilizing  the  joint  significance  test,  as  recommended  by  MacKinnon,  Lockwood, 
Hoffman,  West,  and  Sheets  (2002),  we  found  support  for  mediation.  Based  on  the  significant 
relationship  between  GTR/GIR  and  concept  map  similarity,  and  between  concept  map  similarity 
and  team  performance,  we  concluded  that  team  mental  model  similarity  did  mediate  the 
relationship  of  guided  reflexivity  type  (GTR  versus  GIR)  and  performance.  Similarly,  the 
significant  interaction  of  reflexivity  (GTR/GTR+GIR)  and  storytelling  on  taskwork  mental 
model  similarity,  and  the  relationship  of  taskwork  mental  model  similarity  with  team 
performance,  suggests  the  presence  of  moderated  mediation. 
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Conclusions 

No  study  (to  our  knowledge)  has  researched  the  potential  additive  effects  of  GIR  and  GTR  on 
team  mental  model  similarity.  While  there  has  been  strong  support  for  the  effects  of  team  mental 
model  similarity  on  team  performance  outcomes  (DeChurch  &  Mesmer-Magnus,  2010a,  2010b), 
there  has  not  been  equal  focus  on  the  antecedents  of  team  mental  models  (Mohammed  et  al., 
2010).  Because  it  is  especially  important  for  teams  to  develop  interventions  to  facilitate  shared 
understanding  in  the  absence  of  media-rich  cues,  we  examined  these  relationships  in  distributed 
teams. 

GTR  was  found  to  have  a  stronger  positive  effect  on  team  mental  model  similarity  than  GIR. 
That  is,  being  given  the  opportunity  to  collectively  discuss  their  performance  and  strategize  for 
the  future  improved  teammates’  team  mental  model  similarity  more  so  than  individually 
reflecting  upon  a  video  (regardless  of  whether  containing  a  story  or  more  straightforward 
message)  that  conveyed  the  importance  of  collaboration  and  timing  for  successfully  playing 
NeoCITIES.  Thus,  when  there  is  limited  time  for  reflection,  group  reflexivity  is  preferable. 

When  presented  with  a  storytelling  intervention,  it  is  preferable  to  only  implement  GTR.  In  fact, 
results  suggest  that  storytelling  plus  GTR  yield  the  highest  level  of  team  mental  model  similarity 
out  of  all  the  tested  conditions.  Adding  GIR  immediately  prior  to  GTR  can  actually  decrease 
team  mental  model  similarity.  We  believed  that  having  the  opportunity  to  individually  reflect 
would  allow  team  members  to  better  connect  the  story’s  deep  structure  learning  points  to  the 
NeoCITIES  simulation  in  preparation  for  the  group  reflexivity  session.  However,  it  may  be  that 
having  the  opportunity  to  engage  in  GIR  resulted  in  overconfidence  in  team  members’ 
knowledge  of  how  to  improve  performance,  to  the  point  where  they  did  not  utilize  the  GTR 
‘  intervention  to  its  full  potential. 

The  positive  relationships  of  GTR  and  the  storytelling-reflexivity  interaction  with  team  mental 
model  similarity,  and  the  subsequent  positive  relationship  between  team  mental  model  similarity 
and  team  performance,  suggest  mediated  relationships  (MacKinnon  et  al.,  2002).  Answering  the 
call  for  empirical  work  assessing  multiple  operationalizations  of  team  mental  models  in  the  same 
study  (Mathieu  et  al,  2005;  Mohammed  et  al.,  2010),  both  concept  mapping  and  similarity 
ratings  were  positively  related  to  team  performance  and  evidenced  different,  but  consistent, 
effects  with  storytelling  and  reflexivity.  For  more  information  about  Study  4,  see  Tesler  et  al. 
(2012). 


CONCLUSIONS  ACROSS  FOUR  EXPERIMENTS 

This  grant  project  integrated  research  on  team  cognition ,  temporal  dynamics ,  and  storytelling 
towards  the  goal  of  improving  team  coordination  and  performance  in  distributed  decision  making 
teams. 
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Across  studies,  results  revealed  that  various  types  of  team  cognition  impact  multiple  team 
performance  indices,  demonstrating  that  macrocognition  fosters  team  effectiveness.  Several 
types  of  team  cognition  were  positively  related  to  team  performance,  but  results  differed 
depending  on  the  type  of  team  cognition  (information  sharing,  shared  situation  awareness,  team 
mental  models)  and  the  team  outcome  assessed  (overall  team  performance,  timing).  We 
examined  team  mental  models  through  a  more  sophisticated  lens  than  previous  studies  in  that 
taskwork,  teamwork,  and  temporal  team  mental  models  were  examined  and  both  concept 
mapping  and  paired  comparison  ratings  were  measured  for  temporal  team  mental  models. 
Emphasizing  the  importance  of  these  design  choices,  the  type  of  mental  model  content  and 
operationalization  differentially  affect  team  performance. 

When  team  members  disagree  about  deadlines  and  the  pace  by  which  tasks  should  be  completed, 
coordination  and  performance  problems  can  result.  Therefore,  establishing  and  maintaining 
congruence  in  team  members’  temporal  perceptions  is  a  non-trivial  task.  Infusing  a  temporal 
focus  into  the  study  of  team  cognition  and  team  outcomes  proved  fruitful  in  better  reflecting  the 
time-based  context  of  many  organizational  and  military  teams.  Showing  promise  for  future 
research,  temporal  team  mental  models  were  distinct  from  taskwork  and  teamwork  categories 
and  added  unique  variance  beyond  these  traditional  measures. 

Results  provided  encouraging  evidence  that  storytelling  and  reflexivity  interventions  may  help 
overcome  the  collaborative  obstacles  faced  by  team  members  in  distributed  environments, 
particularly  when  administered  at  the  group  level.  Storytelling  as  an  intervention  did  in  fact 
increase  team  mental  model  similarity  over  a  non-story  format,  which  in  turn  increased  team 
performance.  However,  storytelling  is  most  useful  when  a  team  is  additionally  given  an 
opportunity  to  discuss  and  come  to  a  consensus  on  the  story’s  meaning  while  developing 
strategies  to  improve  future  performance.  Storytelling  may  provide  the  learning  content  but 
reflexivity  provides  the  mechanism  by  which  the  content  can  be  fully  processed.  In  addition, 
allowing  team  members  to  communally  reflect  upon  their  performance  and  strategies  is  more 
effective  than  individual  reflection.  Furthermore,  individual  reflexivity  does  not  provide 
incremental  benefits  when  combined  with  group  reflexivity,  and  can  actually  hurt  outcomes 
when  storytelling  is  also  present. 

SUMMARY  OF  ACCOMPLISHMENTS 

We  ventured  into  non-explored  territory  on  multiple  fronts.  To  our  knowledge,  we  were  the  first 
to  empirically  examine  the  effect  of  planned  storytelling  as  a  team  training  intervention  as  well 
as  to  research  the  potential  additive  effects  of  group  reflexivity  and  individual  reflexivity  on  team 
mental  model  similarity.  In  addition,  we  pioneered  the  operationalization  of  a  temporal  team 
mental  model. 
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We  conducted  four  programmatic  experimental  studies  (ranging  from  71  to  185  three-person 
teams  in  each  study)  examining  the  impact  of  multiple  interventions  (hidden  profile  task, 
storytelling,  group  reflexivity,  individual  reflexivity)  on  multiple  macrocognition  constructs 
(information  sharing,  shared  situation  awareness,  team  mental  models)  and  team  performance. 

We  developed  NeoCITIES  3.0  and  the  setup  of  the  client  server  architecture,  which  was  under 
construction  for  more  than  nine  months,  requiring  over  1 0,000  lines  of  code  and  over  a  month  of 
pilot  testing  the  new  interface.  NeoCITIES  3 .0  allowed  for  a  higher  fidelity  scoring  model  and  a 
modem  technological  infrastructure  that  implemented  a  model  view  controller  framework  for 
modular  interface.  In  addition,  it  allowed  for  an  adaptive  interface  that  was  scalable  and  flexible 
enough  to  accommodate  new  scenario  development.  In  addition,  we  continued  to  upgrade  and 
modify  NeoCITIES  throughout  four  studies,  resulting  in  the  most  recent  3.1  version. 

NeoCITIES  3.0  and  3.1  enabled  the  empirical  investigation  of  under-researched  constructs  such 
as  storytelling  and  temporal  mental  models  as  well  as  integration  across  macrocognitive  concepts 
such  as  situational  awareness  and  information  sharing.  The  NeoCITIES  3.1  interface  and 
scenarios  can  be  useful  to  the  future  design  of  C3  systems  for  ad  hoc  Navy  operations  teams. 

In  terms  of  output,  we  produced  six  publications  (two  book  chapters,  four  published  conference 
proceedings),  fourteen  refereed  conference  presentations,  two  master’s  theses,  and  one 
dissertation.  One  manuscript  is  currently  under  review  with  others  in  preparation.  Conference 
presentations  were  made  to  the  Society  of  Industrial/Organizational  Psychology,  Academy  of 
Management,  Human  Factors  and  Ergonomics  Society,  Interdisciplinary  Network  of  Group 
Researchers,  and  the  International  Science  of  Team  Science.  One  of  the  presentations  was 
featured  as  a  top-rated  poster  at  the  201 1  Society  for  Industrial/Organizational  Psychology 
annual  conference  and  a  finalist  for  the  best  student  submission  award.  The  Appendix  contains  a 
listing  of  grant  output. 
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